{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c256c92c-b35e-4edf-9178-44bfa13339c9",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "# **Presidio + Spark: PII Detection & Anonymization**  \n",
    "\n",
    "This notebook demonstrates how to use Presidio for PII detection and anonymization with Spark in Microsoft Fabric, leveraging SpaCy models for accurate entity recognition.\n",
    "\n",
    "## **Table of Contents**  \n",
    "\n",
    "**1. Small Model Setup & Demo**: Initialize Spark session and set up a lightweight SpaCy-based PII detection model (Small models are not recommended as they might impact accuracy)  \n",
    "**2. Large Model Setup & Demo**: Configure a larger SpaCy model for enhanced PII detection on bigger datasets.  \n",
    "**3. Detect PII Summary**: Run Presidio's analyzer engine on a Spark DataFrame to extract sensitive information.  \n",
    "**4. Anonymize user_query**: Apply Presidio's anonymization engine to mask detected PII while maintaining data utility.  \n",
    "**5. Scale Data (Duplicate Rows)**: Expand datasets by duplicating rows, appending unique IDs, and re-anonymizing.  \n",
    "**6. Write to Delta Table (Optional)**: Persist anonymized data to a Delta table for further analysis.  \n",
    "**7. Scale-Up Testing**: Evaluate performance on larger datasets by duplicating rows and measuring execution time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6534dcc9-85f2-48ea-8a43-881955640bc0",
   "metadata": {
    "collapsed": false,
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:08:05.1368335Z",
       "execution_start_time": "2025-03-05T15:08:02.451639Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "b5c82595-42f2-4cbf-954c-3b0db285b6be",
       "queued_time": "2025-03-05T15:03:47.7410444Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": "2025-03-05T15:03:47.7425316Z",
       "spark_pool": null,
       "state": "finished",
       "statement_id": 5,
       "statement_ids": [
        5
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 5, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.appName(\"PresidioInFabric\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb234d41",
   "metadata": {},
   "source": [
    "## Loading SpaCy Models\n",
    "\n",
    "There are two methods to load SpaCy models:\n",
    "\n",
    "1. **Via the Environment**: For smaller models (e.g., `en_core_web_md` < 300MB) that were included directly in your Fabric environment.\n",
    "   - Note: Small models are not recommended as they might impact accuracy.\n",
    "\n",
    "2. **From the Lakehouse**: For larger models (e.g., `en_core_web_lg` > 300MB), install them within the notebook.\n",
    "   - First, upload the `.whl` file to your Lakehouse\n",
    "   - Then install it using the pip command below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "248ff124-cae2-4db7-a200-b3a104da44e8",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [],
   "source": [
    "%pip install /lakehouse/default/Files/presidio/models/en_core_web_lg-3.8.0-py3-none-any.whl  \n",
    "# Installing the large model from the lakehouse as it exceeds the size limit for custom libraries in the Fabric environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "96a05c63-a589-40f2-9362-d451ce245cbe",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:08:49.0910124Z",
       "execution_start_time": "2025-03-05T15:08:35.026813Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "a9b3fd64-85c9-4b5e-89f1-ed621be63ed7",
       "queued_time": "2025-03-05T15:03:47.7464105Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 13,
       "statement_ids": [
        13
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 13, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pyspark.sql.functions import (\n",
    "    array, lit, explode, col, monotonically_increasing_id, concat\n",
    ")\n",
    "from presidio_analyzer import AnalyzerEngine\n",
    "from presidio_analyzer.nlp_engine import NlpEngineProvider\n",
    "from presidio_anonymizer import AnonymizerEngine\n",
    "from pyspark.sql.functions import udf, col\n",
    "from pyspark.sql.types import ArrayType, StringType\n",
    "from pyspark.sql.functions import pandas_udf, PandasUDFType\n",
    "from presidio_anonymizer.entities import OperatorConfig\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fa5ad734-a655-4ba9-ba78-19610dafb3c8",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:08:49.4716594Z",
       "execution_start_time": "2025-03-05T15:08:49.2433158Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "190e0353-ca38-459f-8cd1-babc7f6655d8",
       "queued_time": "2025-03-05T15:03:47.9619546Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 14,
       "statement_ids": [
        14
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 14, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_duplicates = 5000 # for the scale part\n",
    "csv_path = \"Files/presidio/fabric_sample_data.csv\"\n",
    "is_write_to_delta = True\n",
    "table_namne = \"presidio_demo_table\"\n",
    "partitions_number = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f09acad8-1783-4a05-bf2c-a21e2f9efb10",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## **Small Model Setup & Demo**  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "564b4d58-4fbe-49cf-a23f-0fd1df06eadd",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:08:51.0980594Z",
       "execution_start_time": "2025-03-05T15:08:49.6310635Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "44e595c1-2f9e-4d60-8621-9f0c6aa194c7",
       "queued_time": "2025-03-05T15:03:47.9754566Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 15,
       "statement_ids": [
        15
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 15, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_to_presidio_entity_mapping is missing from configuration, using default\n",
      "low_score_entity_names is missing from configuration, using default\n",
      "labels_to_ignore is missing from configuration, using default\n",
      "Recognizer not added to registry because language is not supported by registry - CreditCardRecognizer supported languages: es, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - CreditCardRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - CreditCardRecognizer supported languages: pl, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - EsNifRecognizer supported languages: es, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - EsNieRecognizer supported languages: es, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItDriverLicenseRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItFiscalCodeRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItVatCodeRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItIdentityCardRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItPassportRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - PlPeselRecognizer supported languages: pl, registry supported languages: en\n",
      "[type: PHONE_NUMBER, start: 46, end: 58, score: 0.75]\n"
     ]
    }
   ],
   "source": [
    "configuration = {\n",
    "    \"nlp_engine_name\": \"spacy\",\n",
    "    \"models\": [\n",
    "        {\"lang_code\": \"en\", \"model_name\": \"en_core_web_md\"},\n",
    "    ]\n",
    "}\n",
    "\n",
    "provider = NlpEngineProvider(nlp_configuration=configuration)\n",
    "nlp_engine = provider.create_engine()\n",
    "\n",
    "small_analyzer = AnalyzerEngine(\n",
    "    nlp_engine=nlp_engine, supported_languages=[\"en\"]\n",
    ")\n",
    "\n",
    "text_to_anonymize = \"His name is Mr. Jones and his phone number is 212-555-5555\"\n",
    "analyzer_results = small_analyzer.analyze(text=text_to_anonymize, entities=[\"PHONE_NUMBER\"], language='en')\n",
    "print(analyzer_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb02c18b-b279-47bb-94c0-b43a9ea531f2",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## **Large Model Setup & Demo**  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1db8d2c7-8697-47bd-9098-ac0ad91f8343",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:08:52.769567Z",
       "execution_start_time": "2025-03-05T15:08:51.2678461Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "a77bc3f3-6a3e-49b5-a8eb-3fc9fcf26995",
       "queued_time": "2025-03-05T15:03:48.0215535Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 16,
       "statement_ids": [
        16
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 16, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_to_presidio_entity_mapping is missing from configuration, using default\n",
      "low_score_entity_names is missing from configuration, using default\n",
      "labels_to_ignore is missing from configuration, using default\n",
      "Recognizer not added to registry because language is not supported by registry - CreditCardRecognizer supported languages: es, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - CreditCardRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - CreditCardRecognizer supported languages: pl, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - EsNifRecognizer supported languages: es, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - EsNieRecognizer supported languages: es, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItDriverLicenseRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItFiscalCodeRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItVatCodeRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItIdentityCardRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - ItPassportRecognizer supported languages: it, registry supported languages: en\n",
      "Recognizer not added to registry because language is not supported by registry - PlPeselRecognizer supported languages: pl, registry supported languages: en\n",
      "[type: PHONE_NUMBER, start: 46, end: 58, score: 0.75]\n"
     ]
    }
   ],
   "source": [
    "configuration = {\n",
    "    \"nlp_engine_name\": \"spacy\",\n",
    "    \"models\": [\n",
    "        {\"lang_code\": \"en\", \"model_name\": \"en_core_web_lg\"},\n",
    "    ]\n",
    "}\n",
    "\n",
    "provider = NlpEngineProvider(nlp_configuration=configuration)\n",
    "nlp_engine = provider.create_engine()\n",
    "\n",
    "analyzer = AnalyzerEngine(\n",
    "    nlp_engine=nlp_engine, supported_languages=[\"en\"]\n",
    ")\n",
    "\n",
    "text_to_anonymize = \"His name is Mr. Jones and his phone number is 212-555-5555\"\n",
    "analyzer_results = analyzer.analyze(text=text_to_anonymize, entities=[\"PHONE_NUMBER\"], language='en')\n",
    "\n",
    "print(analyzer_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de157ec5-c60d-4eb4-87b7-842a0d4ea33f",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## **Load the data and broadcast**\n",
    "Load your data from the CSV file stored in your Lakehouse (e.g., `fabric_sample_data.csv`). This sample data contains various types of PII that we'll detect and anonymize.\n",
    "\n",
    "We broadcast the analyzer and anonymizer objects to make them available to all worker nodes in the Spark cluster, which improves performance for distributed processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ed4110-6de7-4ed6-9e1d-1ee84514c98c",
   "metadata": {
    "collapsed": false,
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:09:13.7585745Z",
       "execution_start_time": "2025-03-05T15:08:52.9319801Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "616eacfa-5a23-4ca2-b06d-e6394d5d77ba",
       "queued_time": "2025-03-05T15:03:48.0380836Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 17,
       "statement_ids": [
        17
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 17, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "b5c48139-db77-44e4-b332-d90c99345f2d",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, b5c48139-db77-44e4-b332-d90c99345f2d)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+\n",
      "|        name|               email|      street|         city|state| non_pii|          user_query|\n",
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+\n",
      "|    John Doe|john.doe@example.com|  123 Elm St|       Dallas|   TX|  abc123|My phone number i...|\n",
      "|  Jane Smith|jane.smith@exampl...|  456 Oak Rd|        Miami|   FL|  xyz789|Please call me at...|\n",
      "| Alice Brown|alice.brown@examp...|  99 Pine Dr|      Seattle|   WA|cust-123|SSN is 987-65-432...|\n",
      "|   Bob Davis|bob.davis@example...|10 Maple Ave|     New York|   NY| info999|Passport number i...|\n",
      "| Carol Jones|carol.jones@examp...|777 Cedar Ln|  Los Angeles|   CA| test111|My phone 333-777-...|\n",
      "| David Green|david.green@examp...|321 Birch St|      Chicago|   IL| npii-01|He said credit ca...|\n",
      "| Emily Clark|emily.clark@examp...|555 Aspen Rd|       Boston|   MA| npii-02|Passport A9876543...|\n",
      "|Frank Wilson|frank.wilson@exam...| 1010 Walnut|      Phoenix|   AZ| npii-03|SSN 666-77-8888 a...|\n",
      "|   Grace Lee|grace.lee@example...|2020 Palm St|      Orlando|   FL| npii-04|Use my card 4000-...|\n",
      "|  Henry King|henry.king@exampl...|  3030 Peach|      Atlanta|   GA| npii-05|NHSNO 0123456789 ...|\n",
      "| Irene Adams|irene.adams@examp...|   4040 Plum|       Denver|   CO| npii-06|SSN 000-11-2222 a...|\n",
      "| Jack Miller|jack.miller@examp...|   5050 Lily|      Houston|   TX| npii-07|My phone 713-555-...|\n",
      "|  Kate Allen|kate.allen@exampl...| 6060 Poplar| Philadelphia|   PA| npii-08|SSN: 123-00-4567,...|\n",
      "|   Leo Clark|leo.clark@example...|   7070 Pine|    Las Vegas|   NV| npii-09|He used card # 44...|\n",
      "|   Mona Reed|mona.reed@example...|    8080 Bay|    San Diego|   CA| npii-10|Call me at 619-55...|\n",
      "|  Nancy Ward|nancy.ward@exampl...|   9090 Moss|San Francisco|   CA| npii-11|Passport is L1234...|\n",
      "|  Oscar Hill|oscar.hill@exampl...|   1111 Hill|     Portland|   OR| npii-12|My card is 4002-1...|\n",
      "|   Paula Ray|paula.ray@example...|  2222 Cliff|       Austin|   TX| npii-13|CC 4012-8888-9999...|\n",
      "|  Ray Foster|ray.foster@exampl...|   3333 Rock|    Nashville|   TN| npii-14|Card # 5555-6666-...|\n",
      "|  Sara White|sara.white@exampl...|  4444 Shell|      Raleigh|   NC| npii-15|SSN 111-22-9999 a...|\n",
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "anonymizer = AnonymizerEngine()\n",
    "broadcasted_analyzer = spark.sparkContext.broadcast(analyzer)\n",
    "broadcasted_anonymizer = spark.sparkContext.broadcast(anonymizer)\n",
    "df = spark.read.format(\"csv\").option(\"header\", \"true\").load(csv_path)\n",
    "display(df)\n",
    "# df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "378754d4-6d25-43c0-98ac-76cf8fe29b2a",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## **Detect PII Summary**\n",
    "\n",
    "This section demonstrates how to:\n",
    "- Instantiate an **AnalyzerEngine** with the SpaCy NLP model\n",
    "- Run PII detection across each row in your DataFrame\n",
    "- Create a summary of detected PII entities by column\n",
    "\n",
    "The function below analyzes each column separately so we know which substring (entity text) belongs to which column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bccee555-6680-49e2-9fe0-92612ece6e2d",
   "metadata": {
    "collapsed": false,
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:11:26.8291003Z",
       "execution_start_time": "2025-03-05T15:11:24.3864644Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "2a2b5823-acbc-4c66-aa45-6a8066802eb9",
       "queued_time": "2025-03-05T15:11:03.4794932Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 22,
       "statement_ids": [
        22
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 22, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "d1fcab6e-5b73-4aa7-9287-b0f4aac44eea",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, d1fcab6e-5b73-4aa7-9287-b0f4aac44eea)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+--------------------+\n",
      "|        name|               email|      street|         city|state| non_pii|          user_query|         pii_summary|\n",
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+--------------------+\n",
      "|    John Doe|john.doe@example.com|  123 Elm St|       Dallas|   TX|  abc123|My phone number i...|{'name': ['PERSON...|\n",
      "|  Jane Smith|jane.smith@exampl...|  456 Oak Rd|        Miami|   FL|  xyz789|Please call me at...|{'name': ['PERSON...|\n",
      "| Alice Brown|alice.brown@examp...|  99 Pine Dr|      Seattle|   WA|cust-123|SSN is 987-65-432...|{'name': ['PERSON...|\n",
      "|   Bob Davis|bob.davis@example...|10 Maple Ave|     New York|   NY| info999|Passport number i...|{'name': ['PERSON...|\n",
      "| Carol Jones|carol.jones@examp...|777 Cedar Ln|  Los Angeles|   CA| test111|My phone 333-777-...|{'name': ['PERSON...|\n",
      "| David Green|david.green@examp...|321 Birch St|      Chicago|   IL| npii-01|He said credit ca...|{'name': ['PERSON...|\n",
      "| Emily Clark|emily.clark@examp...|555 Aspen Rd|       Boston|   MA| npii-02|Passport A9876543...|{'name': ['PERSON...|\n",
      "|Frank Wilson|frank.wilson@exam...| 1010 Walnut|      Phoenix|   AZ| npii-03|SSN 666-77-8888 a...|{'name': ['PERSON...|\n",
      "|   Grace Lee|grace.lee@example...|2020 Palm St|      Orlando|   FL| npii-04|Use my card 4000-...|{'name': ['PERSON...|\n",
      "|  Henry King|henry.king@exampl...|  3030 Peach|      Atlanta|   GA| npii-05|NHSNO 0123456789 ...|{'name': ['PERSON...|\n",
      "| Irene Adams|irene.adams@examp...|   4040 Plum|       Denver|   CO| npii-06|SSN 000-11-2222 a...|{'name': ['PERSON...|\n",
      "| Jack Miller|jack.miller@examp...|   5050 Lily|      Houston|   TX| npii-07|My phone 713-555-...|{'name': ['PERSON...|\n",
      "|  Kate Allen|kate.allen@exampl...| 6060 Poplar| Philadelphia|   PA| npii-08|SSN: 123-00-4567,...|{'name': ['PERSON...|\n",
      "|   Leo Clark|leo.clark@example...|   7070 Pine|    Las Vegas|   NV| npii-09|He used card # 44...|{'name': ['PERSON...|\n",
      "|   Mona Reed|mona.reed@example...|    8080 Bay|    San Diego|   CA| npii-10|Call me at 619-55...|{'name': ['PERSON...|\n",
      "|  Nancy Ward|nancy.ward@exampl...|   9090 Moss|San Francisco|   CA| npii-11|Passport is L1234...|{'name': ['PERSON...|\n",
      "|  Oscar Hill|oscar.hill@exampl...|   1111 Hill|     Portland|   OR| npii-12|My card is 4002-1...|{'name': ['PERSON...|\n",
      "|   Paula Ray|paula.ray@example...|  2222 Cliff|       Austin|   TX| npii-13|CC 4012-8888-9999...|{'name': ['PERSON...|\n",
      "|  Ray Foster|ray.foster@exampl...|   3333 Rock|    Nashville|   TN| npii-14|Card # 5555-6666-...|{'name': ['PERSON...|\n",
      "|  Sara White|sara.white@exampl...|  4444 Shell|      Raleigh|   NC| npii-15|SSN 111-22-9999 a...|{'name': ['PERSON...|\n",
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def detect_pii_in_row(*cols):\n",
    "    \"\"\"\n",
    "    Analyze each column separately so we know which substring (entity text) \n",
    "    belongs to which column. Return a dict {col_name: [ 'ENTITY_TYPE: substring', ... ] }.\n",
    "    \"\"\"\n",
    "    analyzer = broadcasted_analyzer.value\n",
    "    col_names = detect_pii_in_row.col_names\n",
    "    entities_found = {}\n",
    "\n",
    "    for idx, val in enumerate(cols):\n",
    "        if val is None:\n",
    "            continue\n",
    "        column_text = str(val)\n",
    "        results = analyzer.analyze(text=column_text, language=\"en\")\n",
    "\n",
    "        if results:\n",
    "            # Example: [\"PERSON: John Doe\", \"PHONE_NUMBER: 212-555-1111\", ...]\n",
    "            found_entities = []\n",
    "            for res in results:\n",
    "                substring = column_text[res.start:res.end]  # The actual text recognized\n",
    "                entity_str = f\"{res.entity_type}: {substring}\"\n",
    "                found_entities.append(entity_str)\n",
    "            \n",
    "            entities_found[col_names[idx]] = found_entities\n",
    "\n",
    "    # If no PII was detected at all\n",
    "    if not entities_found:\n",
    "        return \"No PII\"\n",
    "    return str(entities_found)\n",
    "\n",
    "detect_pii_in_row.col_names = df.columns\n",
    "detect_pii_udf = udf(detect_pii_in_row, StringType())\n",
    "\n",
    "df_with_pii_summary = df.withColumn(\n",
    "    \"pii_summary\",\n",
    "    detect_pii_udf(*[col(c) for c in df.columns])\n",
    ")\n",
    "\n",
    "display(df_with_pii_summary)\n",
    "# df_with_pii_summary.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b516c51e-e2db-4a66-b431-b6ac4377fb78",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## **Anonymize user_query**\n",
    "\n",
    "This section shows how to:\n",
    "- Use the **AnonymizerEngine** to mask or replace sensitive information\n",
    "- Apply anonymization to specific columns\n",
    "- Preserve the DataFrame structure while removing PII\n",
    "\n",
    "We implement a UDF that detects and anonymizes text in the given column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4adf0050-5079-4bbd-8810-e7a099c6dbc8",
   "metadata": {
    "collapsed": false,
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:11:47.2171767Z",
       "execution_start_time": "2025-03-05T15:11:43.7354063Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "f8850be5-86e7-4379-802e-4006e829f5a3",
       "queued_time": "2025-03-05T15:11:43.5721065Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 23,
       "statement_ids": [
        23
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 23, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "4053ba15-5b54-4ba8-bc55-889fc129cee4",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, 4053ba15-5b54-4ba8-bc55-889fc129cee4)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+--------------------+--------------------+\n",
      "|        name|               email|      street|         city|state| non_pii|          user_query|         pii_summary|     anon_user_query|\n",
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+--------------------+--------------------+\n",
      "|    John Doe|john.doe@example.com|  123 Elm St|       Dallas|   TX|  abc123|My phone number i...|{'name': ['PERSON...|My phone number i...|\n",
      "|  Jane Smith|jane.smith@exampl...|  456 Oak Rd|        Miami|   FL|  xyz789|Please call me at...|{'name': ['PERSON...|Please call me at...|\n",
      "| Alice Brown|alice.brown@examp...|  99 Pine Dr|      Seattle|   WA|cust-123|SSN is 987-65-432...|{'name': ['PERSON...|SSN is <US_ITIN>;...|\n",
      "|   Bob Davis|bob.davis@example...|10 Maple Ave|     New York|   NY| info999|Passport number i...|{'name': ['PERSON...|Passport number i...|\n",
      "| Carol Jones|carol.jones@examp...|777 Cedar Ln|  Los Angeles|   CA| test111|My phone 333-777-...|{'name': ['PERSON...|My phone <PHONE_N...|\n",
      "| David Green|david.green@examp...|321 Birch St|      Chicago|   IL| npii-01|He said credit ca...|{'name': ['PERSON...|He said credit ca...|\n",
      "| Emily Clark|emily.clark@examp...|555 Aspen Rd|       Boston|   MA| npii-02|Passport A9876543...|{'name': ['PERSON...|Passport <US_DRIV...|\n",
      "|Frank Wilson|frank.wilson@exam...| 1010 Walnut|      Phoenix|   AZ| npii-03|SSN 666-77-8888 a...|{'name': ['PERSON...|SSN 666-77-8888 a...|\n",
      "|   Grace Lee|grace.lee@example...|2020 Palm St|      Orlando|   FL| npii-04|Use my card 4000-...|{'name': ['PERSON...|Use my card <CRED...|\n",
      "|  Henry King|henry.king@exampl...|  3030 Peach|      Atlanta|   GA| npii-05|NHSNO 0123456789 ...|{'name': ['PERSON...|NHSNO <UK_NHS> an...|\n",
      "| Irene Adams|irene.adams@examp...|   4040 Plum|       Denver|   CO| npii-06|SSN 000-11-2222 a...|{'name': ['PERSON...|SSN 000-11-2222 a...|\n",
      "| Jack Miller|jack.miller@examp...|   5050 Lily|      Houston|   TX| npii-07|My phone 713-555-...|{'name': ['PERSON...|My phone <PHONE_N...|\n",
      "|  Kate Allen|kate.allen@exampl...| 6060 Poplar| Philadelphia|   PA| npii-08|SSN: 123-00-4567,...|{'name': ['PERSON...|SSN: 123-00-4567,...|\n",
      "|   Leo Clark|leo.clark@example...|   7070 Pine|    Las Vegas|   NV| npii-09|He used card # 44...|{'name': ['PERSON...|He used card # <I...|\n",
      "|   Mona Reed|mona.reed@example...|    8080 Bay|    San Diego|   CA| npii-10|Call me at 619-55...|{'name': ['PERSON...|Call me at <PHONE...|\n",
      "|  Nancy Ward|nancy.ward@exampl...|   9090 Moss|San Francisco|   CA| npii-11|Passport is L1234...|{'name': ['PERSON...|Passport is L1234...|\n",
      "|  Oscar Hill|oscar.hill@exampl...|   1111 Hill|     Portland|   OR| npii-12|My card is 4002-1...|{'name': ['PERSON...|My card is <CREDI...|\n",
      "|   Paula Ray|paula.ray@example...|  2222 Cliff|       Austin|   TX| npii-13|CC 4012-8888-9999...|{'name': ['PERSON...|CC <IN_PAN>9999-1...|\n",
      "|  Ray Foster|ray.foster@exampl...|   3333 Rock|    Nashville|   TN| npii-14|Card # 5555-6666-...|{'name': ['PERSON...|Card # <IN_PAN>77...|\n",
      "|  Sara White|sara.white@exampl...|  4444 Shell|      Raleigh|   NC| npii-15|SSN 111-22-9999 a...|{'name': ['PERSON...|SSN <US_SSN> and ...|\n",
      "+------------+--------------------+------------+-------------+-----+--------+--------------------+--------------------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import udf\n",
    "from pyspark.sql.types import StringType\n",
    "\n",
    "def anonymize_text(text: str) -> str:\n",
    "    \"\"\"\n",
    "    Detect PII in the given text using the large model and replace it with an empty string.\n",
    "    \"\"\"\n",
    "    if text is None:\n",
    "        return None\n",
    "\n",
    "    analyzer = broadcasted_analyzer.value\n",
    "    anonymizer = broadcasted_anonymizer.value\n",
    "    analyze_results = analyzer.analyze(text=text, language=\"en\")\n",
    "    anonymized_result = anonymizer.anonymize(\n",
    "        text=text,\n",
    "        analyzer_results=analyze_results,\n",
    "        operators={\"DEFAULT\": OperatorConfig(\"replace\", {\"new_value\": \"\"})}\n",
    ")\n",
    "    return anonymized_result.text\n",
    "\n",
    "# Registering as a regular PySpark UDF\n",
    "anon_udf = udf(anonymize_text, StringType())\n",
    "\n",
    "df_final = df_with_pii_summary.withColumn(\"anon_user_query\", anon_udf(col(\"user_query\")))\n",
    "\n",
    "display(df_final)\n",
    "# df_final.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e5f6bfb-84bf-4507-b42a-1148022fb370",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## **Scale-Up Testing**\n",
    "\n",
    "This section lets you:\n",
    "- Evaluate performance on larger datasets by duplicating rows\n",
    "- Test how well the solution scales with increasing data volume\n",
    "- Measure execution time for PII detection and anonymization at scale\n",
    "\n",
    "We expand the dataset by creating multiple copies of each row, appending unique IDs, and then re-anonymizing the text to simulate processing a larger dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92e8f682-0e7b-486f-b29e-b265674c9e12",
   "metadata": {
    "collapsed": false,
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:12:33.1069576Z",
       "execution_start_time": "2025-03-05T15:12:24.9454344Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "772c7146-fd2d-4ba9-bb2e-38380261282c",
       "queued_time": "2025-03-05T15:12:24.728969Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 24,
       "statement_ids": [
        24
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 24, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total row number 100000\n"
     ]
    },
    {
     "data": {
      "application/vnd.synapse.widget-view+json": {
       "widget_id": "055e3b9b-76b6-4d30-a98f-43928f3c8603",
       "widget_type": "Synapse.DataFrame"
      },
      "text/plain": [
       "SynapseWidget(Synapse.DataFrame, 055e3b9b-76b6-4d30-a98f-43928f3c8603)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------------------+------------+-----------+-----+-------+--------------------+-----+--------------------+\n",
      "|        name|               email|      street|       city|state|non_pii|          user_query|   id|     anon_user_query|\n",
      "+------------+--------------------+------------+-----------+-----+-------+--------------------+-----+--------------------+\n",
      "|   Leo Clark|leo.clark@example...|   7070 Pine|  Las Vegas|   NV|npii-09|He used card # 44...|65376|He used card # <I...|\n",
      "|  Oscar Hill|oscar.hill@exampl...|   1111 Hill|   Portland|   OR|npii-12|My card is 4002-1...|84635|My card is <CREDI...|\n",
      "| Emily Clark|emily.clark@examp...|555 Aspen Rd|     Boston|   MA|npii-02|Passport A9876543...|34945|Passport <US_DRIV...|\n",
      "|  Oscar Hill|oscar.hill@exampl...|   1111 Hill|   Portland|   OR|npii-12|My card is 4002-1...|81800|My card is <CREDI...|\n",
      "| Irene Adams|irene.adams@examp...|   4040 Plum|     Denver|   CO|npii-06|SSN 000-11-2222 a...|51834|SSN 000-11-2222 a...|\n",
      "| Jack Miller|jack.miller@examp...|   5050 Lily|    Houston|   TX|npii-07|My phone 713-555-...|55111|My phone <PHONE_N...|\n",
      "|Frank Wilson|frank.wilson@exam...| 1010 Walnut|    Phoenix|   AZ|npii-03|SSN 666-77-8888 a...|38326|SSN 666-77-8888 a...|\n",
      "| David Green|david.green@examp...|321 Birch St|    Chicago|   IL|npii-01|He said credit ca...|29259|He said credit ca...|\n",
      "|  Sara White|sara.white@exampl...|  4444 Shell|    Raleigh|   NC|npii-15|SSN 111-22-9999 a...|96171|SSN <US_SSN> and ...|\n",
      "| David Green|david.green@examp...|321 Birch St|    Chicago|   IL|npii-01|He said credit ca...|29983|He said credit ca...|\n",
      "|   Grace Lee|grace.lee@example...|2020 Palm St|    Orlando|   FL|npii-04|Use my card 4000-...|44886|Use my card <CRED...|\n",
      "|   Bob Davis|bob.davis@example...|10 Maple Ave|   New York|   NY|info999|Passport number i...|19619|Passport number i...|\n",
      "|   Bob Davis|bob.davis@example...|10 Maple Ave|   New York|   NY|info999|Passport number i...|17969|Passport number i...|\n",
      "|  Sara White|sara.white@exampl...|  4444 Shell|    Raleigh|   NC|npii-15|SSN 111-22-9999 a...|99185|SSN <US_SSN> and ...|\n",
      "|  Ray Foster|ray.foster@exampl...|   3333 Rock|  Nashville|   TN|npii-14|Card # 5555-6666-...|93467|Card # <IN_PAN>77...|\n",
      "|   Grace Lee|grace.lee@example...|2020 Palm St|    Orlando|   FL|npii-04|Use my card 4000-...|40770|Use my card <CRED...|\n",
      "| Jack Miller|jack.miller@examp...|   5050 Lily|    Houston|   TX|npii-07|My phone 713-555-...|56657|My phone <PHONE_N...|\n",
      "| Carol Jones|carol.jones@examp...|777 Cedar Ln|Los Angeles|   CA|test111|My phone 333-777-...|24322|My phone <PHONE_N...|\n",
      "|   Grace Lee|grace.lee@example...|2020 Palm St|    Orlando|   FL|npii-04|Use my card 4000-...|41338|Use my card <CRED...|\n",
      "|    John Doe|john.doe@example.com|  123 Elm St|     Dallas|   TX| abc123|My phone number i...| 3698|My phone number i...|\n",
      "+------------+--------------------+------------+-----------+-----+-------+--------------------+-----+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_expanded = df.withColumn(\n",
    "    \"duplication_array\",\n",
    "    array([lit(i) for i in range(num_duplicates)])\n",
    ")\n",
    "\n",
    "df_test = df_expanded.withColumn(\"duplicate_id\", explode(col(\"duplication_array\")))\n",
    "\n",
    "df_test = df_test.withColumn(\"id\", monotonically_increasing_id())\n",
    "\n",
    "df_test = df_test.withColumn(\n",
    "    \"user_query\",\n",
    "    concat(col(\"user_query\"), lit(\" - ID: \"), col(\"id\"))\n",
    ")\n",
    "\n",
    "df_test = df_test.drop(\"duplication_array\", \"duplicate_id\")\n",
    "df_test = df_test.repartition(partitions_number) # repartition to show parrallel processing -should be remove/modify to allow high scales.\n",
    "df_test = df_test.withColumn(\"anon_user_query\", anon_udf(col(\"user_query\")))\n",
    "print(f\"total row number {df_test.count()}\") # Number of duplicates X number of rows in the DF\n",
    "display(df_test.limit(partitions_number))\n",
    "# (df_test.limit(partitions_number)).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6605bd6b",
   "metadata": {},
   "source": [
    "## **Write to a Delta Table (Optional)**\n",
    "\n",
    "Optionally save your anonymized data to a Delta table:\n",
    "- Persist results for downstream processing\n",
    "- Make anonymized data available to other Fabric applications\n",
    "- Enable further analysis while maintaining privacy compliance\n",
    "\n",
    "The following code will execute only if `is_write_to_delta` is set to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "612cff00-602e-4cad-8016-efca3ff6d880",
   "metadata": {
    "microsoft": {
     "language": "python",
     "language_group": "synapse_pyspark"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.livy.statement-meta+json": {
       "execution_finish_time": "2025-03-05T15:11:24.2315138Z",
       "execution_start_time": "2025-03-05T15:09:54.048996Z",
       "livy_statement_state": "available",
       "normalized_state": "finished",
       "parent_msg_id": "2e093ed6-54e1-41fe-86c1-68fcd1bfd4a3",
       "queued_time": "2025-03-05T15:03:48.2407357Z",
       "session_id": "1c67db7b-90ca-4222-8845-9aade21d8c79",
       "session_start_time": null,
       "spark_pool": null,
       "state": "finished",
       "statement_id": 21,
       "statement_ids": [
        21
       ]
      },
      "text/plain": [
       "StatementMeta(, 1c67db7b-90ca-4222-8845-9aade21d8c79, 21, Finished, Available, Finished)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if is_write_to_delta:\n",
    "    df_test.write.format(\"delta\").mode(\"overwrite\").saveAsTable(table_namne)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbde0d30",
   "metadata": {},
   "source": [
    "## **Conclusion**\n",
    "\n",
    "You have successfully executed this notebook in Fabric, leveraging Presidio for PII detection/anonymization and Spark for scalable data processing.\n",
    "\n",
    "You can incorporate it into a Fabric pipeline for scheduled runs and integrated workflows. For further customization—like adding detection rules, additional anonymization methods, or advanced Spark configurations—refer to the official Presidio documentation and your Fabric environment's settings.\n",
    "\n",
    "Enjoy building robust PII compliance workflows!"
   ]
  }
 ],
 "metadata": {
  "dependencies": {
   "environment": {
    "environmentId": "02dc06d5-0927-492a-bbd1-a28cf1f0128a",
    "workspaceId": "b94cc731-cd2e-41e4-b428-f6607bb438c4"
   },
   "lakehouse": {
    "default_lakehouse": "d5a01510-2b34-4ab2-8d11-64d572774c4d",
    "default_lakehouse_name": "data",
    "default_lakehouse_workspace_id": "b94cc731-cd2e-41e4-b428-f6607bb438c4",
    "known_lakehouses": [
     {
      "id": "d5a01510-2b34-4ab2-8d11-64d572774c4d"
     }
    ]
   }
  },
  "kernel_info": {
   "name": "synapse_pyspark"
  },
  "kernelspec": {
   "display_name": "Synapse PySpark",
   "language": "Python",
   "name": "synapse_pyspark"
  },
  "language_info": {
   "name": "python"
  },
  "microsoft": {
   "language": "python",
   "language_group": "synapse_pyspark",
   "ms_spell_check": {
    "ms_spell_check_language": "en"
   }
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "spark_compute": {
   "compute_id": "/trident/default",
   "session_options": {
    "conf": {
     "spark.synapse.nbs.session.timeout": "1200000"
    }
   }
  },
  "synapse_widget": {
   "state": {
    "055e3b9b-76b6-4d30-a98f-43928f3c8603": {
     "persist_state": {
      "view": {
       "chartOptions": {
        "aggregationType": "count",
        "binsNumber": 10,
        "categoryFieldKeys": [
         "0"
        ],
        "chartType": "bar",
        "evaluatesOverAllRecords": false,
        "isStacked": false,
        "seriesFieldKeys": [
         "0"
        ],
        "wordFrequency": "-1"
       },
       "tableOptions": {},
       "type": "details",
       "viewOptionsGroup": [
        {
         "tabItems": [
          {
           "key": "0",
           "name": "Table",
           "options": {},
           "type": "table"
          }
         ]
        }
       ]
      }
     },
     "sync_state": {
      "isSummary": false,
      "language": "scala",
      "table": {
       "rows": [
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777. - ID: 65376",
         "7": "65376",
         "8": "He used card # <IN_PAN>4444-4444 and SSN <US_SSN>. - ID: 65376"
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468. - ID: 84635",
         "7": "84635",
         "8": "My card is <CREDIT_CARD> and SSN <US_SSN>. - ID: 84635"
        },
        {
         "0": "Emily Clark",
         "1": "emily.clark@example.com",
         "2": "555 Aspen Rd",
         "3": "Boston",
         "4": "MA",
         "5": "npii-02",
         "6": "Passport A9876543 and SSN 222-33-4444 must be masked. - ID: 34945",
         "7": "34945",
         "8": "Passport <US_DRIVER_LICENSE> and SSN <US_SSN> must be masked. - ID: 34945"
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468. - ID: 81800",
         "7": "81800",
         "8": "My card is <CREDIT_CARD> and SSN <US_SSN>. - ID: 81800"
        },
        {
         "0": "Irene Adams",
         "1": "irene.adams@example.com",
         "2": "4040 Plum",
         "3": "Denver",
         "4": "CO",
         "5": "npii-06",
         "6": "SSN 000-11-2222 and card 6011-1234-5678-9012 must remain private. - ID: 51834",
         "7": "51834",
         "8": "SSN 000-11-2222 and card <IN_PAN>5678-9012 must remain private. - ID: 51834"
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize. - ID: 55111",
         "7": "55111",
         "8": "My phone <PHONE_NUMBER> and passport <US_DRIVER_LICENSE> please anonymize. - ID: 55111"
        },
        {
         "0": "Frank Wilson",
         "1": "frank.wilson@example.com",
         "2": "1010 Walnut",
         "3": "Phoenix",
         "4": "AZ",
         "5": "npii-03",
         "6": "SSN 666-77-8888 and phone 480-555-0000 for reference. - ID: 38326",
         "7": "38326",
         "8": "SSN 666-77-8888 and phone <PHONE_NUMBER> for reference. - ID: 38326"
        },
        {
         "0": "David Green",
         "1": "david.green@example.com",
         "2": "321 Birch St",
         "3": "Chicago",
         "4": "IL",
         "5": "npii-01",
         "6": "He said credit card 4123-4567-8901-2345 and phone 646-555-0123 are on file. - ID: 29259",
         "7": "29259",
         "8": "He said credit card <DATE_TIME> and phone <PHONE_NUMBER> are on file. - ID: 29259"
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both. - ID: 96171",
         "7": "96171",
         "8": "SSN <US_SSN> and phone <PHONE_NUMBER>. <PERSON> both. - ID: 96171"
        },
        {
         "0": "David Green",
         "1": "david.green@example.com",
         "2": "321 Birch St",
         "3": "Chicago",
         "4": "IL",
         "5": "npii-01",
         "6": "He said credit card 4123-4567-8901-2345 and phone 646-555-0123 are on file. - ID: 29983",
         "7": "29983",
         "8": "He said credit card <DATE_TIME> and phone <PHONE_NUMBER> are on file. - ID: 29983"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999. - ID: 44886",
         "7": "44886",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>. - ID: 44886"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222. - ID: 19619",
         "7": "19619",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>. - ID: 19619"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222. - ID: 17969",
         "7": "17969",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>. - ID: 17969"
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both. - ID: 99185",
         "7": "99185",
         "8": "SSN <US_SSN> and phone <PHONE_NUMBER>. <PERSON> both. - ID: 99185"
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000. - ID: 93467",
         "7": "93467",
         "8": "Card # <IN_PAN>7777-8888, phone <PHONE_NUMBER>. - ID: 93467"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999. - ID: 40770",
         "7": "40770",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>. - ID: 40770"
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize. - ID: 56657",
         "7": "56657",
         "8": "My phone <PHONE_NUMBER> and passport <US_DRIVER_LICENSE> please anonymize. - ID: 56657"
        },
        {
         "0": "Carol Jones",
         "1": "carol.jones@example.com",
         "2": "777 Cedar Ln",
         "3": "Los Angeles",
         "4": "CA",
         "5": "test111",
         "6": "My phone 333-777-9090 and SSN 111-22-3333 need to be anonymized. - ID: 24322",
         "7": "24322",
         "8": "My phone <PHONE_NUMBER> and SSN <US_SSN> need to be <IN_PAN>. - ID: 24322"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999. - ID: 41338",
         "7": "41338",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>. - ID: 41338"
        },
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789. - ID: 3698",
         "7": "3698",
         "8": "My phone number is <PHONE_NUMBER> and my SSN is 123-45-6789. - ID: 3698"
        },
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789. - ID: 1524",
         "7": "1524",
         "8": "My phone number is <PHONE_NUMBER> and my SSN is 123-45-6789. - ID: 1524"
        },
        {
         "0": "Irene Adams",
         "1": "irene.adams@example.com",
         "2": "4040 Plum",
         "3": "Denver",
         "4": "CO",
         "5": "npii-06",
         "6": "SSN 000-11-2222 and card 6011-1234-5678-9012 must remain private. - ID: 52579",
         "7": "52579",
         "8": "SSN 000-11-2222 and card <IN_PAN>5678-9012 must remain private. - ID: 52579"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222. - ID: 16935",
         "7": "16935",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>. - ID: <PHONE_NUMBER>"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999. - ID: 43458",
         "7": "43458",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>. - ID: 43458"
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468. - ID: 83699",
         "7": "83699",
         "8": "My card is <CREDIT_CARD> and SSN <US_SSN>. - ID: 83699"
        },
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789. - ID: 3295",
         "7": "3295",
         "8": "My phone number is <PHONE_NUMBER> and my SSN is 123-45-6789. - ID: 3295"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222. - ID: 18669",
         "7": "18669",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>. - ID: 18669"
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000. - ID: 92295",
         "7": "92295",
         "8": "Card # <IN_PAN>7777-8888, phone <PHONE_NUMBER>. - ID: 92295"
        },
        {
         "0": "David Green",
         "1": "david.green@example.com",
         "2": "321 Birch St",
         "3": "Chicago",
         "4": "IL",
         "5": "npii-01",
         "6": "He said credit card 4123-4567-8901-2345 and phone 646-555-0123 are on file. - ID: 27195",
         "7": "27195",
         "8": "He said credit card <DATE_TIME> and phone <PHONE_NUMBER> are on file. - ID: 27195"
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both. - ID: 95392",
         "7": "95392",
         "8": "SSN <US_SSN> and phone <PHONE_NUMBER>. <PERSON> both. - ID: 95392"
        },
        {
         "0": "Paula Ray",
         "1": "paula.ray@example.com",
         "2": "2222 Cliff",
         "3": "Austin",
         "4": "TX",
         "5": "npii-13",
         "6": "CC 4012-8888-9999-1111 and phone 512-555-3333 are on record. - ID: 87740",
         "7": "87740",
         "8": "CC <IN_PAN>9999-1111 and phone <PHONE_NUMBER> are on record. - ID: 87740"
        },
        {
         "0": "Henry King",
         "1": "henry.king@example.com",
         "2": "3030 Peach",
         "3": "Atlanta",
         "4": "GA",
         "5": "npii-05",
         "6": "NHSNO 0123456789 and phone 404-555-1212 are important details. - ID: 49709",
         "7": "49709",
         "8": "NHSNO <UK_NHS> and phone <UK_NHS> are important details. - ID: 49709"
        },
        {
         "0": "Alice Brown",
         "1": "alice.brown@example.com",
         "2": "99 Pine Dr",
         "3": "Seattle",
         "4": "WA",
         "5": "cust-123",
         "6": "SSN is 987-65-4321; credit card: 4444-3333-2222-1111. - ID: 12858",
         "7": "12858",
         "8": "SSN is <US_ITIN>; credit card: <CREDIT_CARD>. - ID: 12858"
        },
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777. - ID: 66726",
         "7": "66726",
         "8": "He used card # <IN_PAN>4444-4444 and SSN <US_SSN>. - ID: 66726"
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468. - ID: 80833",
         "7": "80833",
         "8": "My card is <CREDIT_CARD> and SSN <US_SSN>. - ID: 80833"
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468. - ID: 82644",
         "7": "82644",
         "8": "My card is <CREDIT_CARD> and SSN <US_SSN>. - ID: 82644"
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize. - ID: 56749",
         "7": "56749",
         "8": "My phone <PHONE_NUMBER> and passport <US_DRIVER_LICENSE> please anonymize. - ID: 56749"
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000. - ID: 91868",
         "7": "91868",
         "8": "Card # <IN_PAN>7777-8888, phone <PHONE_NUMBER>. - ID: 91868"
        },
        {
         "0": "Carol Jones",
         "1": "carol.jones@example.com",
         "2": "777 Cedar Ln",
         "3": "Los Angeles",
         "4": "CA",
         "5": "test111",
         "6": "My phone 333-777-9090 and SSN 111-22-3333 need to be anonymized. - ID: 23300",
         "7": "23300",
         "8": "My phone <PHONE_NUMBER> and SSN <US_SSN> need to be <IN_PAN>. - ID: 23300"
        },
        {
         "0": "Mona Reed",
         "1": "mona.reed@example.com",
         "2": "8080 Bay",
         "3": "San Diego",
         "4": "CA",
         "5": "npii-10",
         "6": "Call me at 619-555-9090; also my passport is Z9998887. - ID: 72394",
         "7": "72394",
         "8": "Call me at <PHONE_NUMBER>; also my passport is <US_DRIVER_LICENSE>. - ID: 72394"
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000. - ID: 91946",
         "7": "91946",
         "8": "Card # <IN_PAN>7777-8888, phone <PHONE_NUMBER>. - ID: 91946"
        },
        {
         "0": "Paula Ray",
         "1": "paula.ray@example.com",
         "2": "2222 Cliff",
         "3": "Austin",
         "4": "TX",
         "5": "npii-13",
         "6": "CC 4012-8888-9999-1111 and phone 512-555-3333 are on record. - ID: 88051",
         "7": "88051",
         "8": "CC <IN_PAN>9999-1111 and phone <PHONE_NUMBER> are on record. - ID: 88051"
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both. - ID: 96323",
         "7": "96323",
         "8": "SSN <US_SSN> and phone <PHONE_NUMBER>. <PERSON> both. - ID: 96323"
        },
        {
         "0": "Irene Adams",
         "1": "irene.adams@example.com",
         "2": "4040 Plum",
         "3": "Denver",
         "4": "CO",
         "5": "npii-06",
         "6": "SSN 000-11-2222 and card 6011-1234-5678-9012 must remain private. - ID: 53100",
         "7": "53100",
         "8": "SSN 000-11-2222 and card <IN_PAN>5678-9012 must remain private. - ID: 53100"
        },
        {
         "0": "Henry King",
         "1": "henry.king@example.com",
         "2": "3030 Peach",
         "3": "Atlanta",
         "4": "GA",
         "5": "npii-05",
         "6": "NHSNO 0123456789 and phone 404-555-1212 are important details. - ID: 46630",
         "7": "46630",
         "8": "NHSNO <UK_NHS> and phone <UK_NHS> are important details. - ID: 46630"
        },
        {
         "0": "Kate Allen",
         "1": "kate.allen@example.com",
         "2": "6060 Poplar",
         "3": "Philadelphia",
         "4": "PA",
         "5": "npii-08",
         "6": "SSN: 123-00-4567, phone: 215-555-2222. - ID: 64326",
         "7": "64326",
         "8": "SSN: 123-00-4567, phone: <PHONE_NUMBER>. - ID: 64326"
        },
        {
         "0": "Jane Smith",
         "1": "jane.smith@example.com",
         "2": "456 Oak Rd",
         "3": "Miami",
         "4": "FL",
         "5": "xyz789",
         "6": "Please call me at 555-99-4321 and note my card is 4111-1111-1111-1111. - ID: 9810",
         "7": "9810",
         "8": "Please call me at <US_SSN> and note my card is <CREDIT_CARD>. - ID: 9810"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999. - ID: 40131",
         "7": "40131",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>. - ID: 40131"
        },
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789. - ID: 3638",
         "7": "3638",
         "8": "My phone number is <PHONE_NUMBER> and my SSN is 123-45-6789. - ID: 3638"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222. - ID: 17341",
         "7": "17341",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>. - ID: 17341"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222. - ID: 18793",
         "7": "18793",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>. - ID: 18793"
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize. - ID: 56705",
         "7": "56705",
         "8": "My phone <PHONE_NUMBER> and passport <US_DRIVER_LICENSE> please anonymize. - ID: 56705"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222. - ID: 16529",
         "7": "16529",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>. - ID: 16529"
        },
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789. - ID: 2956",
         "7": "2956",
         "8": "My phone number is <PHONE_NUMBER> and my SSN is 123-45-6789. - ID: 2956"
        },
        {
         "0": "Henry King",
         "1": "henry.king@example.com",
         "2": "3030 Peach",
         "3": "Atlanta",
         "4": "GA",
         "5": "npii-05",
         "6": "NHSNO 0123456789 and phone 404-555-1212 are important details. - ID: 46869",
         "7": "46869",
         "8": "NHSNO <UK_NHS> and phone <UK_NHS> are important details. - ID: 46869"
        },
        {
         "0": "Irene Adams",
         "1": "irene.adams@example.com",
         "2": "4040 Plum",
         "3": "Denver",
         "4": "CO",
         "5": "npii-06",
         "6": "SSN 000-11-2222 and card 6011-1234-5678-9012 must remain private. - ID: 50799",
         "7": "50799",
         "8": "SSN 000-11-2222 and card <IN_PAN>5678-9012 must remain private. - ID: 50799"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999. - ID: 41896",
         "7": "41896",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>. - ID: 41896"
        },
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777. - ID: 69668",
         "7": "69668",
         "8": "He used card # <IN_PAN>4444-4444 and SSN <US_SSN>. - ID: 69668"
        },
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777. - ID: 66594",
         "7": "66594",
         "8": "He used card # <IN_PAN>4444-4444 and SSN <US_SSN>. - ID: 66594"
        },
        {
         "0": "Carol Jones",
         "1": "carol.jones@example.com",
         "2": "777 Cedar Ln",
         "3": "Los Angeles",
         "4": "CA",
         "5": "test111",
         "6": "My phone 333-777-9090 and SSN 111-22-3333 need to be anonymized. - ID: 23308",
         "7": "23308",
         "8": "My phone <PHONE_NUMBER> and SSN <US_SSN> need to be <IN_PAN>. - ID: 23308"
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000. - ID: 90495",
         "7": "90495",
         "8": "Card # <IN_PAN>7777-8888, phone <PHONE_NUMBER>. - ID: 90495"
        },
        {
         "0": "Alice Brown",
         "1": "alice.brown@example.com",
         "2": "99 Pine Dr",
         "3": "Seattle",
         "4": "WA",
         "5": "cust-123",
         "6": "SSN is 987-65-4321; credit card: 4444-3333-2222-1111. - ID: 13221",
         "7": "13221",
         "8": "SSN is <US_ITIN>; credit card: <CREDIT_CARD>. - ID: 13221"
        },
        {
         "0": "David Green",
         "1": "david.green@example.com",
         "2": "321 Birch St",
         "3": "Chicago",
         "4": "IL",
         "5": "npii-01",
         "6": "He said credit card 4123-4567-8901-2345 and phone 646-555-0123 are on file. - ID: 26241",
         "7": "26241",
         "8": "He said credit card <DATE_TIME> and phone <PHONE_NUMBER> are on file. - ID: 26241"
        },
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777. - ID: 67437",
         "7": "67437",
         "8": "He used card # <IN_PAN>4444-4444 and SSN <US_SSN>. - ID: 67437"
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468. - ID: 80444",
         "7": "80444",
         "8": "My card is <CREDIT_CARD> and SSN <US_SSN>. - ID: 80444"
        },
        {
         "0": "Emily Clark",
         "1": "emily.clark@example.com",
         "2": "555 Aspen Rd",
         "3": "Boston",
         "4": "MA",
         "5": "npii-02",
         "6": "Passport A9876543 and SSN 222-33-4444 must be masked. - ID: 32172",
         "7": "32172",
         "8": "Passport <US_DRIVER_LICENSE> and SSN <US_SSN> must be masked. - ID: 32172"
        },
        {
         "0": "Carol Jones",
         "1": "carol.jones@example.com",
         "2": "777 Cedar Ln",
         "3": "Los Angeles",
         "4": "CA",
         "5": "test111",
         "6": "My phone 333-777-9090 and SSN 111-22-3333 need to be anonymized. - ID: 20133",
         "7": "20133",
         "8": "My phone <PHONE_NUMBER> and SSN <US_SSN> need to be <IN_PAN>. - ID: 20133"
        },
        {
         "0": "Paula Ray",
         "1": "paula.ray@example.com",
         "2": "2222 Cliff",
         "3": "Austin",
         "4": "TX",
         "5": "npii-13",
         "6": "CC 4012-8888-9999-1111 and phone 512-555-3333 are on record. - ID: 87488",
         "7": "87488",
         "8": "CC <IN_PAN>9999-1111 and phone <PHONE_NUMBER> are on record. - ID: 87488"
        },
        {
         "0": "Emily Clark",
         "1": "emily.clark@example.com",
         "2": "555 Aspen Rd",
         "3": "Boston",
         "4": "MA",
         "5": "npii-02",
         "6": "Passport A9876543 and SSN 222-33-4444 must be masked. - ID: 34687",
         "7": "34687",
         "8": "Passport <US_DRIVER_LICENSE> and SSN <US_SSN> must be masked. - ID: 34687"
        },
        {
         "0": "Nancy Ward",
         "1": "nancy.ward@example.com",
         "2": "9090 Moss",
         "3": "San Francisco",
         "4": "CA",
         "5": "npii-11",
         "6": "Passport is L1234M567, phone 415-555-1010. - ID: 77268",
         "7": "77268",
         "8": "Passport is L1234M567, phone <PHONE_NUMBER>. - ID: 77268"
        },
        {
         "0": "Frank Wilson",
         "1": "frank.wilson@example.com",
         "2": "1010 Walnut",
         "3": "Phoenix",
         "4": "AZ",
         "5": "npii-03",
         "6": "SSN 666-77-8888 and phone 480-555-0000 for reference. - ID: 35207",
         "7": "35207",
         "8": "SSN 666-77-8888 and phone <PHONE_NUMBER> for reference. - ID: 35207"
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both. - ID: 98801",
         "7": "98801",
         "8": "SSN <US_SSN> and phone <PHONE_NUMBER>. <PERSON> both. - ID: 98801"
        },
        {
         "0": "Nancy Ward",
         "1": "nancy.ward@example.com",
         "2": "9090 Moss",
         "3": "San Francisco",
         "4": "CA",
         "5": "npii-11",
         "6": "Passport is L1234M567, phone 415-555-1010. - ID: 78349",
         "7": "78349",
         "8": "Passport is L1234M567, phone <PHONE_NUMBER>. - ID: 78349"
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000. - ID: 93071",
         "7": "93071",
         "8": "Card # <IN_PAN>7777-8888, phone <PHONE_NUMBER>. - ID: 93071"
        },
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777. - ID: 67643",
         "7": "67643",
         "8": "He used card # <IN_PAN>4444-4444 and SSN <US_SSN>. - ID: 67643"
        },
        {
         "0": "Carol Jones",
         "1": "carol.jones@example.com",
         "2": "777 Cedar Ln",
         "3": "Los Angeles",
         "4": "CA",
         "5": "test111",
         "6": "My phone 333-777-9090 and SSN 111-22-3333 need to be anonymized. - ID: 21085",
         "7": "21085",
         "8": "My phone <PHONE_NUMBER> and SSN <US_SSN> need to be <IN_PAN>. - ID: 21085"
        },
        {
         "0": "Jane Smith",
         "1": "jane.smith@example.com",
         "2": "456 Oak Rd",
         "3": "Miami",
         "4": "FL",
         "5": "xyz789",
         "6": "Please call me at 555-99-4321 and note my card is 4111-1111-1111-1111. - ID: 9165",
         "7": "9165",
         "8": "Please call me at <US_SSN> and note my card is <CREDIT_CARD>. - ID: 9165"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999. - ID: 40166",
         "7": "40166",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>. - ID: 40166"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222. - ID: 15267",
         "7": "15267",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>. - ID: 15267"
        },
        {
         "0": "Mona Reed",
         "1": "mona.reed@example.com",
         "2": "8080 Bay",
         "3": "San Diego",
         "4": "CA",
         "5": "npii-10",
         "6": "Call me at 619-555-9090; also my passport is Z9998887. - ID: 72597",
         "7": "72597",
         "8": "Call me at <PHONE_NUMBER>; also my passport is <US_DRIVER_LICENSE>. - ID: 72597"
        },
        {
         "0": "Alice Brown",
         "1": "alice.brown@example.com",
         "2": "99 Pine Dr",
         "3": "Seattle",
         "4": "WA",
         "5": "cust-123",
         "6": "SSN is 987-65-4321; credit card: 4444-3333-2222-1111. - ID: 10210",
         "7": "10210",
         "8": "SSN is <US_ITIN>; credit card: <CREDIT_CARD>. - ID: 10210"
        },
        {
         "0": "Nancy Ward",
         "1": "nancy.ward@example.com",
         "2": "9090 Moss",
         "3": "San Francisco",
         "4": "CA",
         "5": "npii-11",
         "6": "Passport is L1234M567, phone 415-555-1010. - ID: 78813",
         "7": "78813",
         "8": "Passport is L1234M567, phone <PHONE_NUMBER>. - ID: 78813"
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize. - ID: 57787",
         "7": "57787",
         "8": "My phone <PHONE_NUMBER> and passport <US_DRIVER_LICENSE> please anonymize. - ID: 57787"
        },
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789. - ID: 4456",
         "7": "4456",
         "8": "My phone number is <PHONE_NUMBER> and my SSN is 123-45-6789. - ID: 4456"
        },
        {
         "0": "Emily Clark",
         "1": "emily.clark@example.com",
         "2": "555 Aspen Rd",
         "3": "Boston",
         "4": "MA",
         "5": "npii-02",
         "6": "Passport A9876543 and SSN 222-33-4444 must be masked. - ID: 31018",
         "7": "31018",
         "8": "Passport <US_DRIVER_LICENSE> and SSN <US_SSN> must be masked. - ID: 31018"
        },
        {
         "0": "Henry King",
         "1": "henry.king@example.com",
         "2": "3030 Peach",
         "3": "Atlanta",
         "4": "GA",
         "5": "npii-05",
         "6": "NHSNO 0123456789 and phone 404-555-1212 are important details. - ID: 48677",
         "7": "48677",
         "8": "NHSNO <UK_NHS> and phone <UK_NHS> are important details. - ID: 48677"
        },
        {
         "0": "Henry King",
         "1": "henry.king@example.com",
         "2": "3030 Peach",
         "3": "Atlanta",
         "4": "GA",
         "5": "npii-05",
         "6": "NHSNO 0123456789 and phone 404-555-1212 are important details. - ID: 49272",
         "7": "49272",
         "8": "NHSNO <UK_NHS> and phone <UK_NHS> are important details. - ID: 49272"
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both. - ID: 99844",
         "7": "99844",
         "8": "SSN <US_SSN> and phone <PHONE_NUMBER>. <PERSON> both. - ID: 99844"
        },
        {
         "0": "Emily Clark",
         "1": "emily.clark@example.com",
         "2": "555 Aspen Rd",
         "3": "Boston",
         "4": "MA",
         "5": "npii-02",
         "6": "Passport A9876543 and SSN 222-33-4444 must be masked. - ID: 31733",
         "7": "31733",
         "8": "Passport <US_DRIVER_LICENSE> and SSN <US_SSN> must be masked. - ID: 31733"
        },
        {
         "0": "Paula Ray",
         "1": "paula.ray@example.com",
         "2": "2222 Cliff",
         "3": "Austin",
         "4": "TX",
         "5": "npii-13",
         "6": "CC 4012-8888-9999-1111 and phone 512-555-3333 are on record. - ID: 89594",
         "7": "89594",
         "8": "CC <IN_PAN>9999-1111 and phone <PHONE_NUMBER> are on record. - ID: 89594"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999. - ID: 41053",
         "7": "41053",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>. - ID: 41053"
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize. - ID: 58131",
         "7": "58131",
         "8": "My phone <PHONE_NUMBER> and passport <US_DRIVER_LICENSE> please anonymize. - ID: 58131"
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize. - ID: 56834",
         "7": "56834",
         "8": "My phone <PHONE_NUMBER> and passport <US_DRIVER_LICENSE> please anonymize. - ID: 56834"
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468. - ID: 84907",
         "7": "84907",
         "8": "My card is <CREDIT_CARD> and SSN <US_SSN>. - ID: 84907"
        },
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789. - ID: 4811",
         "7": "4811",
         "8": "My phone number is <PHONE_NUMBER> and my SSN is 123-45-6789. - ID: 4811"
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both. - ID: 99092",
         "7": "99092",
         "8": "SSN <US_SSN> and phone <PHONE_NUMBER>. <PERSON> both. - ID: 99092"
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000. - ID: 90309",
         "7": "90309",
         "8": "Card # <IN_PAN>7777-8888, phone <PHONE_NUMBER>. - ID: 90309"
        },
        {
         "0": "Jane Smith",
         "1": "jane.smith@example.com",
         "2": "456 Oak Rd",
         "3": "Miami",
         "4": "FL",
         "5": "xyz789",
         "6": "Please call me at 555-99-4321 and note my card is 4111-1111-1111-1111. - ID: 9436",
         "7": "9436",
         "8": "Please call me at <US_SSN> and note my card is <CREDIT_CARD>. - ID: 9436"
        },
        {
         "0": "Frank Wilson",
         "1": "frank.wilson@example.com",
         "2": "1010 Walnut",
         "3": "Phoenix",
         "4": "AZ",
         "5": "npii-03",
         "6": "SSN 666-77-8888 and phone 480-555-0000 for reference. - ID: 35717",
         "7": "35717",
         "8": "SSN 666-77-8888 and phone <PHONE_NUMBER> for reference. - ID: 35717"
        },
        {
         "0": "Alice Brown",
         "1": "alice.brown@example.com",
         "2": "99 Pine Dr",
         "3": "Seattle",
         "4": "WA",
         "5": "cust-123",
         "6": "SSN is 987-65-4321; credit card: 4444-3333-2222-1111. - ID: 14429",
         "7": "14429",
         "8": "SSN is <US_ITIN>; credit card: <CREDIT_CARD>. - ID: 14429"
        }
       ],
       "schema": [
        {
         "key": "0",
         "name": "name",
         "type": "string"
        },
        {
         "key": "1",
         "name": "email",
         "type": "string"
        },
        {
         "key": "2",
         "name": "street",
         "type": "string"
        },
        {
         "key": "3",
         "name": "city",
         "type": "string"
        },
        {
         "key": "4",
         "name": "state",
         "type": "string"
        },
        {
         "key": "5",
         "name": "non_pii",
         "type": "string"
        },
        {
         "key": "6",
         "name": "user_query",
         "type": "string"
        },
        {
         "key": "7",
         "name": "id",
         "type": "bigint"
        },
        {
         "key": "8",
         "name": "anon_user_query",
         "type": "string"
        }
       ],
       "truncated": false
      },
      "wranglerEntryContext": {
       "dataframeType": "pyspark"
      }
     },
     "type": "Synapse.DataFrame"
    },
    "4053ba15-5b54-4ba8-bc55-889fc129cee4": {
     "persist_state": {
      "view": {
       "chartOptions": {
        "aggregationType": "count",
        "binsNumber": 10,
        "categoryFieldKeys": [
         "0"
        ],
        "chartType": "bar",
        "evaluatesOverAllRecords": false,
        "isStacked": false,
        "seriesFieldKeys": [
         "0"
        ],
        "wordFrequency": "-1"
       },
       "tableOptions": {},
       "type": "details",
       "viewOptionsGroup": [
        {
         "tabItems": [
          {
           "key": "0",
           "name": "Table",
           "options": {},
           "type": "table"
          }
         ]
        }
       ]
      }
     },
     "sync_state": {
      "isSummary": false,
      "language": "scala",
      "table": {
       "rows": [
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789.",
         "7": "{'name': ['PERSON: John Doe'], 'email': ['EMAIL_ADDRESS: john.doe@example.com', 'URL: john.do', 'URL: example.com'], 'city': ['LOCATION: Dallas'], 'state': ['LOCATION: TX'], 'user_query': ['PHONE_NUMBER: 212-555-1111']}",
         "8": "My phone number is <PHONE_NUMBER> and my SSN is 123-45-6789."
        },
        {
         "0": "Jane Smith",
         "1": "jane.smith@example.com",
         "2": "456 Oak Rd",
         "3": "Miami",
         "4": "FL",
         "5": "xyz789",
         "6": "Please call me at 555-99-4321 and note my card is 4111-1111-1111-1111.",
         "7": "{'name': ['PERSON: Jane Smith'], 'email': ['EMAIL_ADDRESS: jane.smith@example.com', 'URL: jane.sm', 'URL: example.com'], 'city': ['LOCATION: Miami'], 'state': ['LOCATION: FL'], 'user_query': ['CREDIT_CARD: 4111-1111-1111-1111', 'US_SSN: 555-99-4321']}",
         "8": "Please call me at <US_SSN> and note my card is <CREDIT_CARD>."
        },
        {
         "0": "Alice Brown",
         "1": "alice.brown@example.com",
         "2": "99 Pine Dr",
         "3": "Seattle",
         "4": "WA",
         "5": "cust-123",
         "6": "SSN is 987-65-4321; credit card: 4444-3333-2222-1111.",
         "7": "{'name': ['PERSON: Alice Brown'], 'email': ['EMAIL_ADDRESS: alice.brown@example.com', 'URL: alice.br', 'URL: example.com'], 'city': ['LOCATION: Seattle'], 'state': ['LOCATION: WA'], 'user_query': ['CREDIT_CARD: 4444-3333-2222-1111', 'US_ITIN: 987-65-4321']}",
         "8": "SSN is <US_ITIN>; credit card: <CREDIT_CARD>."
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222.",
         "7": "{'name': ['PERSON: Bob Davis'], 'email': ['EMAIL_ADDRESS: bob.davis@example.com', 'URL: example.com'], 'city': ['LOCATION: New York'], 'state': ['LOCATION: NY'], 'non_pii': ['PERSON: info999'], 'user_query': ['PHONE_NUMBER: 212-555-2222', 'US_DRIVER_LICENSE: X1234567']}",
         "8": "Passport number is <US_DRIVER_LICENSE>, phone number <PHONE_NUMBER>."
        },
        {
         "0": "Carol Jones",
         "1": "carol.jones@example.com",
         "2": "777 Cedar Ln",
         "3": "Los Angeles",
         "4": "CA",
         "5": "test111",
         "6": "My phone 333-777-9090 and SSN 111-22-3333 need to be anonymized.",
         "7": "{'name': ['PERSON: Carol Jones'], 'email': ['EMAIL_ADDRESS: carol.jones@example.com', 'URL: carol.jo', 'URL: example.com'], 'city': ['LOCATION: Los Angeles'], 'user_query': ['US_SSN: 111-22-3333', 'PHONE_NUMBER: 333-777-9090']}",
         "8": "My phone <PHONE_NUMBER> and SSN <US_SSN> need to be anonymized."
        },
        {
         "0": "David Green",
         "1": "david.green@example.com",
         "2": "321 Birch St",
         "3": "Chicago",
         "4": "IL",
         "5": "npii-01",
         "6": "He said credit card 4123-4567-8901-2345 and phone 646-555-0123 are on file.",
         "7": "{'name': ['PERSON: David Green'], 'email': ['EMAIL_ADDRESS: david.green@example.com', 'URL: david.green', 'URL: example.com'], 'city': ['LOCATION: Chicago'], 'non_pii': ['DATE_TIME: npii-01'], 'user_query': ['DATE_TIME: 4123-4567-8901-2345', 'PHONE_NUMBER: 646-555-0123', 'IN_PAN: 4123-4567-']}",
         "8": "He said credit card <DATE_TIME> and phone <PHONE_NUMBER> are on file."
        },
        {
         "0": "Emily Clark",
         "1": "emily.clark@example.com",
         "2": "555 Aspen Rd",
         "3": "Boston",
         "4": "MA",
         "5": "npii-02",
         "6": "Passport A9876543 and SSN 222-33-4444 must be masked.",
         "7": "{'name': ['PERSON: Emily Clark'], 'email': ['EMAIL_ADDRESS: emily.clark@example.com', 'URL: emily.cl', 'URL: example.com'], 'city': ['LOCATION: Boston'], 'non_pii': ['DATE_TIME: npii-02'], 'user_query': ['US_SSN: 222-33-4444', 'US_DRIVER_LICENSE: A9876543']}",
         "8": "Passport <US_DRIVER_LICENSE> and SSN <US_SSN> must be masked."
        },
        {
         "0": "Frank Wilson",
         "1": "frank.wilson@example.com",
         "2": "1010 Walnut",
         "3": "Phoenix",
         "4": "AZ",
         "5": "npii-03",
         "6": "SSN 666-77-8888 and phone 480-555-0000 for reference.",
         "7": "{'name': ['PERSON: Frank Wilson'], 'email': ['EMAIL_ADDRESS: frank.wilson@example.com', 'URL: example.com'], 'street': ['LOCATION: Walnut'], 'city': ['LOCATION: Phoenix'], 'state': ['LOCATION: AZ'], 'user_query': ['PHONE_NUMBER: 480-555-0000']}",
         "8": "SSN 666-77-8888 and phone <PHONE_NUMBER> for reference."
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999.",
         "7": "{'name': ['PERSON: Grace Lee'], 'email': ['EMAIL_ADDRESS: grace.lee@example.com', 'URL: example.com'], 'street': ['DATE_TIME: 2020'], 'city': ['LOCATION: Orlando'], 'state': ['LOCATION: FL'], 'non_pii': ['DATE_TIME: npii-04'], 'user_query': ['CREDIT_CARD: 4000-0000-0000-0002', 'PHONE_NUMBER: 407-555-9999', 'IN_PAN: 4000-0000-']}",
         "8": "Use my card <CREDIT_CARD> and phone <PHONE_NUMBER>."
        },
        {
         "0": "Henry King",
         "1": "henry.king@example.com",
         "2": "3030 Peach",
         "3": "Atlanta",
         "4": "GA",
         "5": "npii-05",
         "6": "NHSNO 0123456789 and phone 404-555-1212 are important details.",
         "7": "{'name': ['PERSON: Henry King'], 'email': ['EMAIL_ADDRESS: henry.king@example.com', 'URL: henry.ki', 'URL: example.com'], 'city': ['LOCATION: Atlanta'], 'non_pii': ['DATE_TIME: npii-05'], 'user_query': ['UK_NHS: 0123456789', 'UK_NHS: 404-555-1212', 'PHONE_NUMBER: 404-555-1212', 'IN_PAN: 0123456789', 'US_BANK_NUMBER: 0123456789', 'US_DRIVER_LICENSE: 0123456789']}",
         "8": "NHSNO <UK_NHS> and phone <UK_NHS> are important details."
        },
        {
         "0": "Irene Adams",
         "1": "irene.adams@example.com",
         "2": "4040 Plum",
         "3": "Denver",
         "4": "CO",
         "5": "npii-06",
         "6": "SSN 000-11-2222 and card 6011-1234-5678-9012 must remain private.",
         "7": "{'name': ['PERSON: Irene Adams'], 'email': ['EMAIL_ADDRESS: irene.adams@example.com', 'URL: irene.ad', 'URL: example.com'], 'city': ['LOCATION: Denver'], 'non_pii': ['DATE_TIME: npii-06'], 'user_query': ['IN_PAN: 6011-1234-']}",
         "8": "SSN 000-11-2222 and card <IN_PAN>5678-9012 must remain private."
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize.",
         "7": "{'name': ['PERSON: Jack Miller'], 'email': ['EMAIL_ADDRESS: jack.miller@example.com', 'URL: jack.mil', 'URL: example.com'], 'city': ['LOCATION: Houston'], 'state': ['LOCATION: TX'], 'non_pii': ['DATE_TIME: npii-07'], 'user_query': ['PHONE_NUMBER: 713-555-0101', 'US_DRIVER_LICENSE: B1234567']}",
         "8": "My phone <PHONE_NUMBER> and passport <US_DRIVER_LICENSE> please anonymize."
        },
        {
         "0": "Kate Allen",
         "1": "kate.allen@example.com",
         "2": "6060 Poplar",
         "3": "Philadelphia",
         "4": "PA",
         "5": "npii-08",
         "6": "SSN: 123-00-4567, phone: 215-555-2222.",
         "7": "{'name': ['PERSON: Kate Allen'], 'email': ['EMAIL_ADDRESS: kate.allen@example.com', 'URL: kate.al', 'URL: example.com'], 'city': ['LOCATION: Philadelphia'], 'state': ['LOCATION: PA'], 'non_pii': ['DATE_TIME: npii-08'], 'user_query': ['PHONE_NUMBER: 215-555-2222']}",
         "8": "SSN: 123-00-4567, phone: <PHONE_NUMBER>."
        },
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777.",
         "7": "{'name': ['PERSON: Leo Clark'], 'email': ['EMAIL_ADDRESS: leo.clark@example.com', 'URL: leo.cl', 'URL: example.com'], 'city': ['LOCATION: Las Vegas'], 'non_pii': ['DATE_TIME: npii-09'], 'user_query': ['US_SSN: 555-66-7777', 'IN_PAN: 4444-4444-']}",
         "8": "He used card # <IN_PAN>4444-4444 and SSN <US_SSN>."
        },
        {
         "0": "Mona Reed",
         "1": "mona.reed@example.com",
         "2": "8080 Bay",
         "3": "San Diego",
         "4": "CA",
         "5": "npii-10",
         "6": "Call me at 619-555-9090; also my passport is Z9998887.",
         "7": "{'name': ['PERSON: Mona Reed'], 'email': ['EMAIL_ADDRESS: mona.reed@example.com', 'URL: mona.re', 'URL: example.com'], 'city': ['LOCATION: San Diego'], 'non_pii': ['DATE_TIME: npii-10'], 'user_query': ['PHONE_NUMBER: 619-555-9090', 'US_DRIVER_LICENSE: Z9998887']}",
         "8": "Call me at <PHONE_NUMBER>; also my passport is <US_DRIVER_LICENSE>."
        },
        {
         "0": "Nancy Ward",
         "1": "nancy.ward@example.com",
         "2": "9090 Moss",
         "3": "San Francisco",
         "4": "CA",
         "5": "npii-11",
         "6": "Passport is L1234M567, phone 415-555-1010.",
         "7": "{'name': ['PERSON: Nancy Ward'], 'email': ['EMAIL_ADDRESS: nancy.ward@example.com', 'URL: example.com'], 'city': ['LOCATION: San Francisco'], 'non_pii': ['DATE_TIME: npii-11'], 'user_query': ['PHONE_NUMBER: 415-555-1010']}",
         "8": "Passport is L1234M567, phone <PHONE_NUMBER>."
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468.",
         "7": "{'name': ['PERSON: Oscar Hill'], 'email': ['EMAIL_ADDRESS: oscar.hill@example.com', 'URL: example.com'], 'city': ['LOCATION: Portland'], 'non_pii': ['DATE_TIME: npii-12'], 'user_query': ['CREDIT_CARD: 4002-1234-5678-9999', 'US_SSN: 135-79-2468', 'IN_PAN: 4002-1234-']}",
         "8": "My card is <CREDIT_CARD> and SSN <US_SSN>."
        },
        {
         "0": "Paula Ray",
         "1": "paula.ray@example.com",
         "2": "2222 Cliff",
         "3": "Austin",
         "4": "TX",
         "5": "npii-13",
         "6": "CC 4012-8888-9999-1111 and phone 512-555-3333 are on record.",
         "7": "{'name': ['PERSON: Paula Ray'], 'email': ['EMAIL_ADDRESS: paula.ray@example.com', 'URL: example.com'], 'city': ['PERSON: Austin'], 'state': ['LOCATION: TX'], 'user_query': ['PHONE_NUMBER: 512-555-3333', 'IN_PAN: 4012-8888-']}",
         "8": "CC <IN_PAN>9999-1111 and phone <PHONE_NUMBER> are on record."
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000.",
         "7": "{'name': ['PERSON: Ray Foster'], 'email': ['EMAIL_ADDRESS: ray.foster@example.com', 'URL: ray.fo', 'URL: example.com'], 'city': ['LOCATION: Nashville'], 'state': ['LOCATION: TN'], 'user_query': ['PHONE_NUMBER: 615-555-1000', 'IN_PAN: 5555-6666-']}",
         "8": "Card # <IN_PAN>7777-8888, phone <PHONE_NUMBER>."
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both.",
         "7": "{'name': ['PERSON: Sara White'], 'email': ['EMAIL_ADDRESS: sara.white@example.com', 'URL: example.com'], 'state': ['LOCATION: NC'], 'non_pii': ['DATE_TIME: npii-15'], 'user_query': ['US_SSN: 111-22-9999', 'PERSON: Anonymize', 'PHONE_NUMBER: 919-555-5678']}",
         "8": "SSN <US_SSN> and phone <PHONE_NUMBER>. <PERSON> both."
        }
       ],
       "schema": [
        {
         "key": "0",
         "name": "name",
         "type": "string"
        },
        {
         "key": "1",
         "name": "email",
         "type": "string"
        },
        {
         "key": "2",
         "name": "street",
         "type": "string"
        },
        {
         "key": "3",
         "name": "city",
         "type": "string"
        },
        {
         "key": "4",
         "name": "state",
         "type": "string"
        },
        {
         "key": "5",
         "name": "non_pii",
         "type": "string"
        },
        {
         "key": "6",
         "name": "user_query",
         "type": "string"
        },
        {
         "key": "7",
         "name": "pii_summary",
         "type": "string"
        },
        {
         "key": "8",
         "name": "anon_user_query",
         "type": "string"
        }
       ],
       "truncated": false
      },
      "wranglerEntryContext": {
       "candidateVariableNames": [
        "df_final"
       ],
       "dataframeType": "pyspark"
      }
     },
     "type": "Synapse.DataFrame"
    },
    "b5c48139-db77-44e4-b332-d90c99345f2d": {
     "persist_state": {
      "view": {
       "chartOptions": {
        "aggregationType": "count",
        "binsNumber": 10,
        "categoryFieldKeys": [
         "0"
        ],
        "chartType": "bar",
        "evaluatesOverAllRecords": false,
        "isStacked": false,
        "seriesFieldKeys": [
         "0"
        ],
        "wordFrequency": "-1"
       },
       "tableOptions": {},
       "type": "details",
       "viewOptionsGroup": [
        {
         "tabItems": [
          {
           "key": "0",
           "name": "Table",
           "options": {},
           "type": "table"
          }
         ]
        }
       ]
      }
     },
     "sync_state": {
      "isSummary": false,
      "language": "scala",
      "table": {
       "rows": [
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789."
        },
        {
         "0": "Jane Smith",
         "1": "jane.smith@example.com",
         "2": "456 Oak Rd",
         "3": "Miami",
         "4": "FL",
         "5": "xyz789",
         "6": "Please call me at 555-99-4321 and note my card is 4111-1111-1111-1111."
        },
        {
         "0": "Alice Brown",
         "1": "alice.brown@example.com",
         "2": "99 Pine Dr",
         "3": "Seattle",
         "4": "WA",
         "5": "cust-123",
         "6": "SSN is 987-65-4321; credit card: 4444-3333-2222-1111."
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222."
        },
        {
         "0": "Carol Jones",
         "1": "carol.jones@example.com",
         "2": "777 Cedar Ln",
         "3": "Los Angeles",
         "4": "CA",
         "5": "test111",
         "6": "My phone 333-777-9090 and SSN 111-22-3333 need to be anonymized."
        },
        {
         "0": "David Green",
         "1": "david.green@example.com",
         "2": "321 Birch St",
         "3": "Chicago",
         "4": "IL",
         "5": "npii-01",
         "6": "He said credit card 4123-4567-8901-2345 and phone 646-555-0123 are on file."
        },
        {
         "0": "Emily Clark",
         "1": "emily.clark@example.com",
         "2": "555 Aspen Rd",
         "3": "Boston",
         "4": "MA",
         "5": "npii-02",
         "6": "Passport A9876543 and SSN 222-33-4444 must be masked."
        },
        {
         "0": "Frank Wilson",
         "1": "frank.wilson@example.com",
         "2": "1010 Walnut",
         "3": "Phoenix",
         "4": "AZ",
         "5": "npii-03",
         "6": "SSN 666-77-8888 and phone 480-555-0000 for reference."
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999."
        },
        {
         "0": "Henry King",
         "1": "henry.king@example.com",
         "2": "3030 Peach",
         "3": "Atlanta",
         "4": "GA",
         "5": "npii-05",
         "6": "NHSNO 0123456789 and phone 404-555-1212 are important details."
        },
        {
         "0": "Irene Adams",
         "1": "irene.adams@example.com",
         "2": "4040 Plum",
         "3": "Denver",
         "4": "CO",
         "5": "npii-06",
         "6": "SSN 000-11-2222 and card 6011-1234-5678-9012 must remain private."
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize."
        },
        {
         "0": "Kate Allen",
         "1": "kate.allen@example.com",
         "2": "6060 Poplar",
         "3": "Philadelphia",
         "4": "PA",
         "5": "npii-08",
         "6": "SSN: 123-00-4567, phone: 215-555-2222."
        },
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777."
        },
        {
         "0": "Mona Reed",
         "1": "mona.reed@example.com",
         "2": "8080 Bay",
         "3": "San Diego",
         "4": "CA",
         "5": "npii-10",
         "6": "Call me at 619-555-9090; also my passport is Z9998887."
        },
        {
         "0": "Nancy Ward",
         "1": "nancy.ward@example.com",
         "2": "9090 Moss",
         "3": "San Francisco",
         "4": "CA",
         "5": "npii-11",
         "6": "Passport is L1234M567, phone 415-555-1010."
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468."
        },
        {
         "0": "Paula Ray",
         "1": "paula.ray@example.com",
         "2": "2222 Cliff",
         "3": "Austin",
         "4": "TX",
         "5": "npii-13",
         "6": "CC 4012-8888-9999-1111 and phone 512-555-3333 are on record."
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000."
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both."
        }
       ],
       "schema": [
        {
         "key": "0",
         "name": "name",
         "type": "string"
        },
        {
         "key": "1",
         "name": "email",
         "type": "string"
        },
        {
         "key": "2",
         "name": "street",
         "type": "string"
        },
        {
         "key": "3",
         "name": "city",
         "type": "string"
        },
        {
         "key": "4",
         "name": "state",
         "type": "string"
        },
        {
         "key": "5",
         "name": "non_pii",
         "type": "string"
        },
        {
         "key": "6",
         "name": "user_query",
         "type": "string"
        }
       ],
       "truncated": false
      },
      "wranglerEntryContext": {
       "candidateVariableNames": [
        "df"
       ],
       "dataframeType": "pyspark"
      }
     },
     "type": "Synapse.DataFrame"
    },
    "d1fcab6e-5b73-4aa7-9287-b0f4aac44eea": {
     "persist_state": {
      "view": {
       "chartOptions": {
        "aggregationType": "count",
        "binsNumber": 10,
        "categoryFieldKeys": [
         "0"
        ],
        "chartType": "bar",
        "evaluatesOverAllRecords": false,
        "isStacked": false,
        "seriesFieldKeys": [
         "0"
        ],
        "wordFrequency": "-1"
       },
       "tableOptions": {},
       "type": "details",
       "viewOptionsGroup": [
        {
         "tabItems": [
          {
           "key": "0",
           "name": "Table",
           "options": {},
           "type": "table"
          }
         ]
        }
       ]
      }
     },
     "sync_state": {
      "isSummary": false,
      "language": "scala",
      "table": {
       "rows": [
        {
         "0": "John Doe",
         "1": "john.doe@example.com",
         "2": "123 Elm St",
         "3": "Dallas",
         "4": "TX",
         "5": "abc123",
         "6": "My phone number is 212-555-1111 and my SSN is 123-45-6789.",
         "7": "{'name': ['PERSON: John Doe'], 'email': ['EMAIL_ADDRESS: john.doe@example.com', 'URL: john.do', 'URL: example.com'], 'city': ['LOCATION: Dallas'], 'state': ['LOCATION: TX'], 'user_query': ['PHONE_NUMBER: 212-555-1111']}"
        },
        {
         "0": "Jane Smith",
         "1": "jane.smith@example.com",
         "2": "456 Oak Rd",
         "3": "Miami",
         "4": "FL",
         "5": "xyz789",
         "6": "Please call me at 555-99-4321 and note my card is 4111-1111-1111-1111.",
         "7": "{'name': ['PERSON: Jane Smith'], 'email': ['EMAIL_ADDRESS: jane.smith@example.com', 'URL: jane.sm', 'URL: example.com'], 'city': ['LOCATION: Miami'], 'state': ['LOCATION: FL'], 'user_query': ['CREDIT_CARD: 4111-1111-1111-1111', 'US_SSN: 555-99-4321']}"
        },
        {
         "0": "Alice Brown",
         "1": "alice.brown@example.com",
         "2": "99 Pine Dr",
         "3": "Seattle",
         "4": "WA",
         "5": "cust-123",
         "6": "SSN is 987-65-4321; credit card: 4444-3333-2222-1111.",
         "7": "{'name': ['PERSON: Alice Brown'], 'email': ['EMAIL_ADDRESS: alice.brown@example.com', 'URL: alice.br', 'URL: example.com'], 'city': ['LOCATION: Seattle'], 'state': ['LOCATION: WA'], 'user_query': ['CREDIT_CARD: 4444-3333-2222-1111', 'US_ITIN: 987-65-4321']}"
        },
        {
         "0": "Bob Davis",
         "1": "bob.davis@example.com",
         "2": "10 Maple Ave",
         "3": "New York",
         "4": "NY",
         "5": "info999",
         "6": "Passport number is X1234567, phone number 212-555-2222.",
         "7": "{'name': ['PERSON: Bob Davis'], 'email': ['EMAIL_ADDRESS: bob.davis@example.com', 'URL: example.com'], 'city': ['LOCATION: New York'], 'state': ['LOCATION: NY'], 'non_pii': ['PERSON: info999'], 'user_query': ['PHONE_NUMBER: 212-555-2222', 'US_DRIVER_LICENSE: X1234567']}"
        },
        {
         "0": "Carol Jones",
         "1": "carol.jones@example.com",
         "2": "777 Cedar Ln",
         "3": "Los Angeles",
         "4": "CA",
         "5": "test111",
         "6": "My phone 333-777-9090 and SSN 111-22-3333 need to be anonymized.",
         "7": "{'name': ['PERSON: Carol Jones'], 'email': ['EMAIL_ADDRESS: carol.jones@example.com', 'URL: carol.jo', 'URL: example.com'], 'city': ['LOCATION: Los Angeles'], 'user_query': ['US_SSN: 111-22-3333', 'PHONE_NUMBER: 333-777-9090']}"
        },
        {
         "0": "David Green",
         "1": "david.green@example.com",
         "2": "321 Birch St",
         "3": "Chicago",
         "4": "IL",
         "5": "npii-01",
         "6": "He said credit card 4123-4567-8901-2345 and phone 646-555-0123 are on file.",
         "7": "{'name': ['PERSON: David Green'], 'email': ['EMAIL_ADDRESS: david.green@example.com', 'URL: david.green', 'URL: example.com'], 'city': ['LOCATION: Chicago'], 'non_pii': ['DATE_TIME: npii-01'], 'user_query': ['DATE_TIME: 4123-4567-8901-2345', 'PHONE_NUMBER: 646-555-0123', 'IN_PAN: 4123-4567-']}"
        },
        {
         "0": "Emily Clark",
         "1": "emily.clark@example.com",
         "2": "555 Aspen Rd",
         "3": "Boston",
         "4": "MA",
         "5": "npii-02",
         "6": "Passport A9876543 and SSN 222-33-4444 must be masked.",
         "7": "{'name': ['PERSON: Emily Clark'], 'email': ['EMAIL_ADDRESS: emily.clark@example.com', 'URL: emily.cl', 'URL: example.com'], 'city': ['LOCATION: Boston'], 'non_pii': ['DATE_TIME: npii-02'], 'user_query': ['US_SSN: 222-33-4444', 'US_DRIVER_LICENSE: A9876543']}"
        },
        {
         "0": "Frank Wilson",
         "1": "frank.wilson@example.com",
         "2": "1010 Walnut",
         "3": "Phoenix",
         "4": "AZ",
         "5": "npii-03",
         "6": "SSN 666-77-8888 and phone 480-555-0000 for reference.",
         "7": "{'name': ['PERSON: Frank Wilson'], 'email': ['EMAIL_ADDRESS: frank.wilson@example.com', 'URL: example.com'], 'street': ['LOCATION: Walnut'], 'city': ['LOCATION: Phoenix'], 'state': ['LOCATION: AZ'], 'user_query': ['PHONE_NUMBER: 480-555-0000']}"
        },
        {
         "0": "Grace Lee",
         "1": "grace.lee@example.com",
         "2": "2020 Palm St",
         "3": "Orlando",
         "4": "FL",
         "5": "npii-04",
         "6": "Use my card 4000-0000-0000-0002 and phone 407-555-9999.",
         "7": "{'name': ['PERSON: Grace Lee'], 'email': ['EMAIL_ADDRESS: grace.lee@example.com', 'URL: example.com'], 'street': ['DATE_TIME: 2020'], 'city': ['LOCATION: Orlando'], 'state': ['LOCATION: FL'], 'non_pii': ['DATE_TIME: npii-04'], 'user_query': ['CREDIT_CARD: 4000-0000-0000-0002', 'PHONE_NUMBER: 407-555-9999', 'IN_PAN: 4000-0000-']}"
        },
        {
         "0": "Henry King",
         "1": "henry.king@example.com",
         "2": "3030 Peach",
         "3": "Atlanta",
         "4": "GA",
         "5": "npii-05",
         "6": "NHSNO 0123456789 and phone 404-555-1212 are important details.",
         "7": "{'name': ['PERSON: Henry King'], 'email': ['EMAIL_ADDRESS: henry.king@example.com', 'URL: henry.ki', 'URL: example.com'], 'city': ['LOCATION: Atlanta'], 'non_pii': ['DATE_TIME: npii-05'], 'user_query': ['UK_NHS: 0123456789', 'UK_NHS: 404-555-1212', 'PHONE_NUMBER: 404-555-1212', 'IN_PAN: 0123456789', 'US_BANK_NUMBER: 0123456789', 'US_DRIVER_LICENSE: 0123456789']}"
        },
        {
         "0": "Irene Adams",
         "1": "irene.adams@example.com",
         "2": "4040 Plum",
         "3": "Denver",
         "4": "CO",
         "5": "npii-06",
         "6": "SSN 000-11-2222 and card 6011-1234-5678-9012 must remain private.",
         "7": "{'name': ['PERSON: Irene Adams'], 'email': ['EMAIL_ADDRESS: irene.adams@example.com', 'URL: irene.ad', 'URL: example.com'], 'city': ['LOCATION: Denver'], 'non_pii': ['DATE_TIME: npii-06'], 'user_query': ['IN_PAN: 6011-1234-']}"
        },
        {
         "0": "Jack Miller",
         "1": "jack.miller@example.com",
         "2": "5050 Lily",
         "3": "Houston",
         "4": "TX",
         "5": "npii-07",
         "6": "My phone 713-555-0101 and passport B1234567 please anonymize.",
         "7": "{'name': ['PERSON: Jack Miller'], 'email': ['EMAIL_ADDRESS: jack.miller@example.com', 'URL: jack.mil', 'URL: example.com'], 'city': ['LOCATION: Houston'], 'state': ['LOCATION: TX'], 'non_pii': ['DATE_TIME: npii-07'], 'user_query': ['PHONE_NUMBER: 713-555-0101', 'US_DRIVER_LICENSE: B1234567']}"
        },
        {
         "0": "Kate Allen",
         "1": "kate.allen@example.com",
         "2": "6060 Poplar",
         "3": "Philadelphia",
         "4": "PA",
         "5": "npii-08",
         "6": "SSN: 123-00-4567, phone: 215-555-2222.",
         "7": "{'name': ['PERSON: Kate Allen'], 'email': ['EMAIL_ADDRESS: kate.allen@example.com', 'URL: kate.al', 'URL: example.com'], 'city': ['LOCATION: Philadelphia'], 'state': ['LOCATION: PA'], 'non_pii': ['DATE_TIME: npii-08'], 'user_query': ['PHONE_NUMBER: 215-555-2222']}"
        },
        {
         "0": "Leo Clark",
         "1": "leo.clark@example.com",
         "2": "7070 Pine",
         "3": "Las Vegas",
         "4": "NV",
         "5": "npii-09",
         "6": "He used card # 4444-4444-4444-4444 and SSN 555-66-7777.",
         "7": "{'name': ['PERSON: Leo Clark'], 'email': ['EMAIL_ADDRESS: leo.clark@example.com', 'URL: leo.cl', 'URL: example.com'], 'city': ['LOCATION: Las Vegas'], 'non_pii': ['DATE_TIME: npii-09'], 'user_query': ['US_SSN: 555-66-7777', 'IN_PAN: 4444-4444-']}"
        },
        {
         "0": "Mona Reed",
         "1": "mona.reed@example.com",
         "2": "8080 Bay",
         "3": "San Diego",
         "4": "CA",
         "5": "npii-10",
         "6": "Call me at 619-555-9090; also my passport is Z9998887.",
         "7": "{'name': ['PERSON: Mona Reed'], 'email': ['EMAIL_ADDRESS: mona.reed@example.com', 'URL: mona.re', 'URL: example.com'], 'city': ['LOCATION: San Diego'], 'non_pii': ['DATE_TIME: npii-10'], 'user_query': ['PHONE_NUMBER: 619-555-9090', 'US_DRIVER_LICENSE: Z9998887']}"
        },
        {
         "0": "Nancy Ward",
         "1": "nancy.ward@example.com",
         "2": "9090 Moss",
         "3": "San Francisco",
         "4": "CA",
         "5": "npii-11",
         "6": "Passport is L1234M567, phone 415-555-1010.",
         "7": "{'name': ['PERSON: Nancy Ward'], 'email': ['EMAIL_ADDRESS: nancy.ward@example.com', 'URL: example.com'], 'city': ['LOCATION: San Francisco'], 'non_pii': ['DATE_TIME: npii-11'], 'user_query': ['PHONE_NUMBER: 415-555-1010']}"
        },
        {
         "0": "Oscar Hill",
         "1": "oscar.hill@example.com",
         "2": "1111 Hill",
         "3": "Portland",
         "4": "OR",
         "5": "npii-12",
         "6": "My card is 4002-1234-5678-9999 and SSN 135-79-2468.",
         "7": "{'name': ['PERSON: Oscar Hill'], 'email': ['EMAIL_ADDRESS: oscar.hill@example.com', 'URL: example.com'], 'city': ['LOCATION: Portland'], 'non_pii': ['DATE_TIME: npii-12'], 'user_query': ['CREDIT_CARD: 4002-1234-5678-9999', 'US_SSN: 135-79-2468', 'IN_PAN: 4002-1234-']}"
        },
        {
         "0": "Paula Ray",
         "1": "paula.ray@example.com",
         "2": "2222 Cliff",
         "3": "Austin",
         "4": "TX",
         "5": "npii-13",
         "6": "CC 4012-8888-9999-1111 and phone 512-555-3333 are on record.",
         "7": "{'name': ['PERSON: Paula Ray'], 'email': ['EMAIL_ADDRESS: paula.ray@example.com', 'URL: example.com'], 'city': ['PERSON: Austin'], 'state': ['LOCATION: TX'], 'user_query': ['PHONE_NUMBER: 512-555-3333', 'IN_PAN: 4012-8888-']}"
        },
        {
         "0": "Ray Foster",
         "1": "ray.foster@example.com",
         "2": "3333 Rock",
         "3": "Nashville",
         "4": "TN",
         "5": "npii-14",
         "6": "Card # 5555-6666-7777-8888, phone 615-555-1000.",
         "7": "{'name': ['PERSON: Ray Foster'], 'email': ['EMAIL_ADDRESS: ray.foster@example.com', 'URL: ray.fo', 'URL: example.com'], 'city': ['LOCATION: Nashville'], 'state': ['LOCATION: TN'], 'user_query': ['PHONE_NUMBER: 615-555-1000', 'IN_PAN: 5555-6666-']}"
        },
        {
         "0": "Sara White",
         "1": "sara.white@example.com",
         "2": "4444 Shell",
         "3": "Raleigh",
         "4": "NC",
         "5": "npii-15",
         "6": "SSN 111-22-9999 and phone 919-555-5678. Anonymize both.",
         "7": "{'name': ['PERSON: Sara White'], 'email': ['EMAIL_ADDRESS: sara.white@example.com', 'URL: example.com'], 'state': ['LOCATION: NC'], 'non_pii': ['DATE_TIME: npii-15'], 'user_query': ['US_SSN: 111-22-9999', 'PERSON: Anonymize', 'PHONE_NUMBER: 919-555-5678']}"
        }
       ],
       "schema": [
        {
         "key": "0",
         "name": "name",
         "type": "string"
        },
        {
         "key": "1",
         "name": "email",
         "type": "string"
        },
        {
         "key": "2",
         "name": "street",
         "type": "string"
        },
        {
         "key": "3",
         "name": "city",
         "type": "string"
        },
        {
         "key": "4",
         "name": "state",
         "type": "string"
        },
        {
         "key": "5",
         "name": "non_pii",
         "type": "string"
        },
        {
         "key": "6",
         "name": "user_query",
         "type": "string"
        },
        {
         "key": "7",
         "name": "pii_summary",
         "type": "string"
        }
       ],
       "truncated": false
      },
      "wranglerEntryContext": {
       "candidateVariableNames": [
        "df_with_pii_summary"
       ],
       "dataframeType": "pyspark"
      }
     },
     "type": "Synapse.DataFrame"
    }
   },
   "version": "0.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
